Grand Challenges in Virtual Environments

In his 1999 article that reviewed the state of virtual reality (VR) Prof. Fred Brooks Jr. noted that at that time the field had made great technical advances over the previous 5 years. Brooks (1999) wrote “I think our technology has crossed over the pass – VR that used to almost work now barely works. VR is now really real.” What is the state today? Needless to say 15 years later, it is the case that not only does VR “really work” but that it has become a commonplace tool in many areas of science, technology, psychological therapy, medical rehabilitation, marketing, and industry, and is surely about to become commonplace in the home. Let us very briefly reprise the major technologies reviewed by Brooks (1999). On the display side, projection systems such as Caves (Cruz-Neira et al., 1992) have advanced to very high resolution, based on multiple panel displays with automatic and seamless image alignment (Brown et al., 2005; Defanti et al., 2011). Headmounted displays (HMDs) have improved dramatically – presenting wide field-ofview high-resolution images, with very recent advances toward lightweight, low cost, and consumer oriented devices with wide field-of-view and acceptable resolution. Beyond specialized devices there is the promise of yet a further development with already usable HMDs made from 3D printable plastic frames and cheap lenses that house a Smartphone (Olson et al., 2011; Hoberman et al., 2012; Steed and Julier, 2013). Devices for virtual and augmented reality displays typically require the participant to wear specialized glasses. Autostereo displays obviate the need for this but offer a limited continuous field-of-view (Holliman et al., 2011). An important grand challenge in the display area is the provision of high quality full field-of-view stereodisplays that do not require special glasses, where there is a seamless blend between reality and VR. Steps are being made in this direction (Hilliges et al., 2012) and also there is the development of such displays based on the low cost consumer devices (Maimone et al., 2012). The other side of the equation to display is tracking. In recent years whole body tracking has become a relatively low cost product enabling real-time motion capture. But just as with HMDs, the arrival of consumer oriented tracking systems and depth cameras, marketed for computer games, is likely to revolutionize how real-time full body tracking and probably head-tracking are likely to develop. Head and body tracking over wide areas with low latency and high accuracy remains a significant challenge. One thing is clear from the above – VR is moving to the home. However, the vast majority of the studies that assess various aspects of the responses of people to VR experiences typically involve very short one-off exposure times – a notable exception being (Steed et al., 2003). Due to the move to the consumer market this situation obviously will and must change – although simulator sickness may still be a significant problem. If people in their millions will be using these systems for hours a week, we really need to understand the impact of this on their lives and the resulting social impact, as well as seeing this as an opportunity to carry out massive experimental studies of scientific interest that also inform the technology. The above gives the impression that VR is defined by devices and associated software. However, it is more useful to consider VR conceptually as a technological system that can precisely substitute a person’s sensory input and transform the meaning of their motor outputs with reference to an exactly knowable alternate reality (“knowable” to distinguish from dreams or hallucinogenic experiences). In this view motor actions and sensory input are not separable. In order to perceive it is necessary to act – to move and position the body, head, eyes, ears, nose, and end-effectors, in active perception. There are implicit rules that we acquire that integrate perception and action, referred to as sensorimotor contingencies (O’Regan and Noë, 2001; Noë, 2004). The affordance of natural sensorimotor contingencies for perception by a VR system is a key to the generation of the fundamental illusion that people typically experience in an immersive VR – the illusion of “presence,” “being there,” or “place illusion (PI)” (Sanchez-Vives and Slater, 2005; Slater, 2009). Sensorimotor contingencies refer to acts of perception by the participant that change his or her sensory stream. However, there will typically also be autonomous changes or events in the sensory stream that are not caused by the actions of the participant. When such events, not under the control of the participant, nevertheless contingently occur in response to participant actions this can give rise to a second illusion – that the perceived events are really happening (even though they are known for sure to be occurring in a fake reality). This “plausibility illusion (Psi)” refers therefore to the dynamic aspects of the virtual environment, whereas PI can occur in a completely static environment. Psi also requires that if the situation being simulated is one that could be encountered in reality, then it must meet at least minimal expectations as to how that reality functions. This heralds the fact that achieving Psi is a significantly greater challenge than